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WWC Study Reports are intended to support decision making; neither the What Works Clearinghouse 
(WWC) nor the U.S. Department of Education endorses any interventions. No single Study Report should 
be used as a basis for making policy decisions because (1) few studies are designed and implemented 
flawlessly and (2) all studies are tested on a limited number of participants, using a limited number of 
outcomes, at a limited number of times, so generalizing from one study to any context is very difficult. 

To highlight these issues, the WWC Study Reports describe in detail the specifics of each study, focusing 
primarily on studies that provide the best evidence of effects (randomized controlled trials). Systematic 
reviews of the evidence will be conducted to summarize the results of the individual studies. 

See the WWC Brief Study Report (PDF) for a summary of this study. 



Topic: Curriculum-Based Interventions for Increasing K-12 Math Achievement — 
Middle School 



Intervention: Connected Mathematics Project 
Research Design: Quasi-Experimental Design with Matching 

Study Rating: Q 



Date Released: March 4, 2005 

Summary of Results: Riordan and Noyce (2001 ) report that schools that used the Connected 

Mathematics Project (CMP) for two to three years had greater gains in 
math achievement overall than those in the comparison group. Riordan 
and Noyce do not report whether this difference was statistically 
significant. However, t tests calculated by the WWC using data provided 
by Riordan and Noyce indicate that the difference is not statistically 
significant. In additional analyses, Riordan and Noyce compared 
performances on four mathematics topics covered by the outcome 
measure and found that students in the CMP schools scored statistically 
significantly higher in all of these areas. 



= Meets Evidence Standards = Meets Evidence Standards with Reservations Q = Does Not Meet Evidence Standards 



The What Works Clearinghouse ( www.whatworks.ed.gov) was established in 2002 by the U.S. 
Department of Education’s Institute of Education Sciences to provide educators, policymakers, 
researchers, and the public with a central and trusted source of scientific evidence of what works in 
education. Please email all questions and comments to info@whatworks.ed.qov . The What Works 
Clearinghouse is administered by the U.S. Department of Education through a contract to a joint 
venture of the American Institutes for Research and the Campbell Collaboration . 
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Intervention: Connected 
Mathematics Project 

Operational Features 

Between 1991 and 1997, the Connected 
Mathematics Project (CMP) received funding 
from the National Science Foundation to 
develop a mathematics curriculum for middle 
school (grades 6 through 8). The CMP, 
developed at Michigan State University, is 
reported to adhere to the National Council of 
Teachers of Mathematics (NCTM) Curriculum 
and Evaluation Standards. 

The key feature of the CMP is that it is a 
problem-centered curriculum that provides skills 
practice. The CMP is a comprehensive 
mathematics curriculum that has 24 units, with 
eight units at each of three grade levels. Each 
unit contains four to seven investigations. Each 
investigation in turn contains one to five 
problem sets and ends with a mathematical 
reflection activity. 

Support for implementing the CMP curriculum 
includes professional development prior to 
implementation and follow-up professional 
development each summer after 
implementation. In addition, pilot sites in this 
study provided regular meetings with their 
teachers throughout the school year. 

In this study, students in the intervention group 
used the CMP curriculum and students in the 
comparison group used a mix of traditional 
math curricula. 

People, Settings, and Timing 

The CMP is designed to be used in grades 6 
through 8. The students in this study were all in 
8th grade in regular education and were drawn 
from schools in Massachusetts. Riordan and 
Noyce (2001) do not indicate where in 
Massachusetts the schools were located or 
whether the schools were in a variety of 
settings. The schools were in relatively 
privileged areas, with low rates of students 
participating in the free and reduced-price lunch 
program. The students are described as 
predominantly white, but no information was 
provided about the gender ratio or the 



percentage of students who had limited 
proficiency in English. 

Cost Information 

Cost information is not reported in this study. 

Intended Duration 

Each year’ s CMP curriculum consists of eight 
units, and the developer describes the units as 
cumulative, with each unit building on the 
previous one. Full implementation consists of 
covering all eight units during the school year, 
although the developer indicates that schools 
can phase in the program slowly and work up to 
having the full number of units per year. 

Scientific Rationale 

The key hypothesis of this study is that students 
in standards-based curricula will perform better 
on standardized tests than students in traditional 
curricula, and schools with standards-based 
curricula will show greater increases in math 
achievement over time. Standards-based 
curricula are aligned with the standards created 
by the National Council of Teachers of 
Mathematics and are based on the idea that 
students learn better through their own 
investigations, which contrasts with traditional 
curricula that emphasize memorization and 
lectures. 

Overview of the Study 

Purpose 

The central question tested in this study was 
whether or not a difference would exist in the 
average standardized test scores in math at 
schools that had implemented the CMP 
curriculum when compared with the average 
scores at schools that used a mix of traditional 
math curricula. Riordan and Noyce hypothesize 
that students using the CMP curriculum will 
perform better on state standardized tests than 
students using the traditional curriculum. 

Intervention Fidelity 

According to the report, the curriculum was 
implemented differently from the way the 
developer intended. Riordan and Noyce indicate 
that schools had to have implemented at least 1 1 
CMP units in grades 6 through 8 (by 1998-99) 
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to be included in the intervention group. 
Although schools could vary the number of 
units implemented per grade, the schools were 
unable to implement the full eight units that the 
CMP has available for each grade. 

Outcome Measures 

The primary outcome measure for this study 
was the Massachusetts Comprehensive 
Assessment System (MCAS). This standardized 
test was the only posttest measure used in the 
study. Riordan and Noyce do not indicate when 
the outcome measure was administered. 

Research Design 

To identify schools using the CMP curriculum, 
Riordan and Noyce obtained a list of middle 
schools using this curriculum from the Center 
for the Enhancement of Science and 
Mathematics Education (CESAME). The list 
was compared to the results from the 1999 
Mathematics, Science, and Technology Survey. 
To be included in the study, a school had to 
appear on the CESAME list and be reported by 
the survey or the publisher as using the 
curriculum. Because the CMP is a modular 
program, schools can implement a few modules 
at a time. Telephone interviews were conducted 
with school personnel to determine how many 
CMP units they were implementing. Through 
this process, 21 of 408 middle schools in 
Massachusetts were selected as the intervention 
group. One of these schools had implemented 
the CMP curriculum earlier than the others and 
was analyzed separately by Riordan and Noyce. 
This school is not included in this report. 

The comparison group of schools was 
determined through a series of steps. First, 
Riordan and Noyce performed a regression 
analysis to determine the strongest predictors of 
achievement. Then the intervention and 
comparison schools were matched on the basis 
of (1) their average Massachusetts Educational 
Assessment Program (MEAP) scores in the last 
year preceding the curriculum implementation 1 



1 The Commonwealth of Massachusetts legislated in 1993 
a requirement that new assessments should be more in- 
line with new state standards. One result was the 
Massachusetts Comprehensive Assessment System 



and (2) the percentage of students receiving free 
and/or reduced-price lunch. The textbooks most 
commonly used by the comparison group were 
published by Heath, Addison-Wesley, Prentice 
Hall, and Houghton Mifflin. 

Participant Flow 

Riordan and Noyce added schools to the 
comparison group because the schools in the 
intervention and comparison groups differed in 
size. Riordan and Noyce do not indicate that any 
attrition from the sample occurred. Participant 
flow is presented in Figure 1 . 



(MCAS), which was first administered in 1998. For this 
reason, the pretest is a different assessment from the 
posttest. 
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Figure 1. Participant Flow (Schools) 




Purposeful Exclusions 
from Analysis: 

5 schools excluded from 
analysis 









Lost to Analysis: 
Participants lost to 
analysis 






Analysis Sample 



20 



) 



30 



) 



Note. One intervention school implemented the CMP curriculum earlier than the others and was analyzed separately by Riordan 
and Noyce. This school and its matched comparison schools were excluded from the analysis at the school level. NR = not 
reported. 
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Reference Periods 

This study covers the time period from 1996 to 
1999. The group of schools discussed in this 
report had implemented the CMP curriculum for 
two to three years by the time of the posttest. 

Baseline Data 

At the baseline, there were no significant 
differences in the 1996 MEAP scores of the 
intervention and comparison groups. Both the 
intervention and comparison schools had 
average scores that were above the state mean. 
About 10% of students in the intervention group 
participated in the free and reduced-price lunch 
program, compared with 1 1% in the comparison 
group. This difference was not statistically 
significant. About 81% of students in the 
intervention group were white, while about 89% 
of the students in the comparison schools were 
white. This difference was statistically 
significant. 

Statistical Methods 

Two types of analyses were done: one at the 
student level and one at the school level. This 
WWC Study Report focuses only on the school- 
level analyses, because assignment occurred at 
the school level. 

In their study, Riordan and Noyce compared Z 
scores at the school level to assess the relative 
change in average scores. Riordan and Noyce 
also compared the percentage of possible points 
scored, on average, in the four mathematical 
topic areas: number sense, patterns and 
functions, geometry, and statistics. Only regular 
education students were included in the 
analyses. 

Outcomes and Estimation 

Table 1 presents the effects of the CMP 
curriculum on math achievement, as reported by 
Riordan and Noyce. In a follow-up request by 
the WWC, Riordan and Noyce provided school- 
level math achievement means in addition to the 
Z scores they reported in their study. The WWC 
performed t tests on this information, which 
indicate that the difference was not statistically 
significant. 



Table 2 presents the results of Riordan and 
Noyce’s analysis of differences between the 
scores of the CMP schools and comparison 
schools across four mathematics topic areas on 
the MCAS. Riordan and Noyce only present 
information about whether the CMP schools 
scored significantly better or worse than the 
comparison schools, not the actual average 
scores for the schools. Because the magnitude of 
the differences was small, Riordan and Noyce 
chose to represent the differences as: (+) when 
the CMP schools scored significantly better than 
the comparison schools, (-) when the CMP 
schools scored significantly worse than the 
comparison schools, and (0) when there was no 
significant difference. 

Although there are many legitimate ways to 
calculate effects, the WWC uses the same 
analysis strategies for computing effect sizes for 
all studies, to facilitate comparison and 
aggregation of effects across different studies. 
The WWC reanalyzed the data to produce 
estimates of effect sizes. (See Table 3 and 
Figure 2.) Table 3 presents unadjusted effect 
size calculations for the average of the scores 
for all schools. In computing effect sizes, the 
WWC created a standardized mean difference 
estimate using the pooled standard deviation of 
the groups as the denominator. Only means and 
standard deviations were used to calculate the 
effect sizes. 
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Table 1. Impact Reported by Riordan and Noyce (2001): Posttest on 
Math Achievement 



Posttest on Math 


Intervention 


Comparison 


Intervention Group 


Achievement 


Group ( SD ) 


Group (SD) 


Estimated Effect Size 


Mean MCAS 








Math Score, 1999 


238.20 (9.08) 


233.85 (10.57) 


NR 



Note. MCAS = Massachusetts Comprehensive Assessment Program. NR = not reported. 



Table 2. Differences Reported by Riordan and Noyce (2001) Between 
CMP Schools and Comparison Schools on Four Mathematics Topics 



Mathematics Topic 


Difference 


Significance 


Number Sense 


+ 


p < .001 


Patterns and Functions 


+ 


p < .001 


Geometry 


+ 


p < .001 


Statistics 


+ 


p < .001 



Table 3. Impact Calculated by WWC: Posttest on Math Achievement 



Posttest on 
Math 

Achievement 



Intervention 
Group Mean 
Score 
(n = 20 
schools) 



Comparison 
Group Mean 
Score 
(« = 30 
schools) 



Estimated Effect 3 
Mean 

Difference Effect Size b 



CI c p Value d 



Mean MCAS 238.20 233.85 4.3 .43 +/- .57 > .05 

Math Score, 

1999 



Note. MCAS = Massachusetts Comprehensive Assessment Program. 

a Effect shown both as the mean difference between the intervention and comparison groups and as a relative effect size, 
which is equal to the ratio of the mean difference to the standard deviation of the outcome measure. 

b All estimates of effect size are based on the d-index. 
c Confidence Interval around the effect size estimate. 
d Reflects significance level of the effect size estimate. 
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Figure 2. Impact Calculated by Riordan and Noyce (2001 ) a : Posttest 
Math Scores 




Connected Mathematics Comparison Mean Difference 

Project (n = 30 schools) 

(n = 20 schools) 



Note. Riordan and Noyce did not report whether the difference between the groups was significant. 
MCAS = Massachusetts Comprehensive Assessment Program. 

a Confidence intervals were calculated by the WWC. 



Intervention Developer Contact 
Information 

Riordan and Noyce do not package, distribute, 
or provide technical assistance on the CMP. The 
CMP curricula materials are distributed by: 

Pearson Prentice Hall, Inc. 

1-800-848-9500 

www.phschool.com/math/cmp/index.html 

Related Studies 

See reports on other studies of the Connected 
Mathematics Project. 

See reports on other studies of Middle School 
Math curricula. 



Report Production 

Date created: March 4, 2005 

Topic area reviewed under: Curriculum-Based 
Interventions for Increasing K-12 Math 
Achievement — Middle School 
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WWC Study Ratings 3 : Riordan & Noyce (2001) 



Causal Validity: Meets WWC Evidence Standards with Reservations, Quasi-Experimental Design 

with Matching 

Both assignment and matching were performed at the school level. Schools were first matched based on their average scores 
on the Massachusetts Educational Assessment Program test for the year prior to the introduction of the Connected 
Mathematics Project (CMP). Schools were also matched on the percentage of students in each school that were eligible for 
free and reduced-price lunch. Riordan and Noyce do not report any extraneous events that might have confounded the 
intervention’s effects. 



Other Study 
Characteristics 


Study Rating 


Study-Specific Information 


Intervention Fidelity 


• 


The CMP curriculum is well designed and implemented and meets the definition of 
middle school math, but the intervention was not implemented in the way it was 
intended. None of the schools implemented the full eight units the CMP has available 
for each grade. The schools also differed from each other in the number of units they 
implemented. 


Outcome Measures 


• • 


One math achievement outcome measure, the Massachusetts Comprehensive 
Assessment Program, was used in this study. There was evidence that this outcome was 
sufficiently reliable and that it was properly aligned to the intervention. Riordan and 
Noyce do not report when the outcome measure was administered. 


People, Settings, and 
Timing 


• 


Although some important characteristics of the target population were captured in this 
study, many were not. Riordan and Noyce present information at the school level about 
the overall percentage of white students in the schools and the percentage of students 
receiving free and reduced-price lunch in the schools. Riordan and Noyce do not report 
variation by such characteristics as gender and limited English proficiency. Not enough 
detail was given in the article to determine whether there was variation in settings. A 
reasonable range of content areas was included in the outcome measure. 


Testing within 
Subgroups 


• 


The effectiveness of the intervention was tested across the sample of schools but not 
within important subgroups. 


Analysis 


• 


The unit of assignment (school) matched the units of statistical analysis (school). The 
statistical assumptions necessary for analysis were met. The sample of schools was 
small, therefore the precision of statistical estimates is limited. 


Statistical Reporting 


• • 


Riordan and Noyce provide the unadjusted means and standard deviations of the 
achievement posttest. Riordan and Noyce also report sample sizes for each group; thus 
an effect estimate could be calculated. 



Note. • • Fully meets criteria; • Meets minimum criteria; X Does not meet criteria. 

a For more information on the criteria used to rate this study, see the WWC Evidence Standards: Middle School Math. 
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